Metrics  for  Analyzing  Quantifiable  Differentiation  of  Designs  with  Varying 

Integrity  for  Hardware  Assurance 


Adam  G.  Kimura\  Steven  B.  Bibyk',  Brian  P.  Dupaix1,  Matthew  J.  Casto2,  Gregory  L.  Creech 1 

^he  Ohio  State  University  -  Department  of  Electrical  and  Computer  Engineering  (Columbus,  OH) 

2Wright  Patterson  Air  Force  Research  Labs  (Dayton,  OH) 

Contact  Author  Email:  kimura.1 1  @osu.edu 


Abstract  —  This  work  proposes  an  approach  to  quantifying 
the  integrity  of  a  questionable  design  by  parsing  the  design 
into  characteristic  sub-domains:  Logical  Equivalence,  Signal 
Activity  Rate,  Functional  Correctness,  Structural  Architecture, 
and  Power  Consumption.  Measurement  techniques  are 
reviewed  for  each  domain  which  quantify  deviation  of  the 
actual  design  away  from  the  expected  profiles.  A  novel  method 
for  quantifying  the  quality  of  reference  used  for  expected 
profiles  is  also  proposed.  Expected  profiles  can  incorporate  a 
level  of  overdesign.  Finally,  the  Design  Integrity  measuring 
techniques  are  applied  to  five  Test  Article  (TA)  cases  that 
showed  Error  2  TA  to  have  the  lowest  integrity  of  2.95/5  and 
the  untampered  TA  containing  the  highest  integrity  of  5.00/5. 

Keywords  —  Trojan;  integrity;  trust;  quantify;  hardware; 
assurance;  verification;  metrics;  reference,  quality;  profile 

I.  Introduction 

A.  The  Rising  Concern  of  Hardware  Trust 

As  Integrated  Circuit  (IC)  chips  continue  to  advance  in 
complexity,  economics  and  time-to-market  pressures  are  driving 
hardware  developers  into  distributed  design  processes  and 
complex  supply  chains.  This  has  led  to  more  opportunistic  points 
in  the  design  and  manufacturing  flow  for  error  insertion  by 
adversarial  or  dishonest  agents  inside  a  supplier.  A  hardware  error 
is  defined  as  any  construct  that  causes  deviation  from  the  intended 
specification.  Hardware  errors  are  typically  categorized  as  either 
faults  or  hardware  Trojans.  Hardware  Trojans  are  inserted  into 
the  design  with  malicious  intent  to  compromise  a  design’s 
functionality  and  reliability.  Other  aims  of  hardware  Trojans 
could  be  for  granting  control  to  an  adversary  for  monitoring  or 
stealing  information.  A  fault  is  a  quality  control  occurrence 
usually  caused  by  poor  fabrication  processes;  however  faults  are 
not  typically  malicious  in  nature. 

With  the  globalization  of  the  IC  industry,  hardware 
untrustworthiness  (i.e.  the  concern  for  hardware  error  insertion) 
has  become  a  growing  issue  as  the  Internet  of  Things  continues  to 
expand  [1].  The  rising  concern  for  hardware  trust  has  therefore 
led  to  the  emergence  of  a  new  field  of  research  to  address  these 
concerns  -  Trusted  Microelectronics. 


B.  The  Need  for  Trust  Metrics 

Developing  a  portfolio  of  metrics  to  quantify  aspects  of  Trust 
such  as  design  integrity  and  vulnerability  are  critical  components 
to  Trusted  Microelectronics.  Trust  Metrics  for  quantifying  design 
integrity  could  provide  measurable  insight  into  how  closely  the 
fabricated  hardware  from  an  untrusted  foundry  matches  the 
original  design  by  providing  a  distance  measure  for  how  far  it 
deviated  from  it.  Hardware  vulnerability  metrics  could  also  be 
integrated  into  Computer- Automated  Design  (CAD)  toolsets  in 
order  to  grant  quantifiable  insight  into  various  vulnerability 
mitigation  strategies  for  improving  design  security.  In  addition, 
as  standards  for  Trusted  Microelectronics  are  developed,  trust 
figures  of  merit  may  be  utilized  as  a  framework  for  benchmarking 
Trusted  Part  certifications. 

Previous  work  conducted  in  Trust  Metric  development  has 
focused  on  measures  at  the  supplier  level  of  abstraction  [2]  [3]. 
By  quantifying  the  integrity  of  questionable  hardware  at  the 
design  level,  one  gains  the  granularity  to  address  the  trust  concerns 
on  a  part-by-part  basis.  This  work  presents  techniques  to  measure 
the  integrity  of  hardware  at  the  design  level  and  arrives  at  a  metric 
that  can  be  used  to  gauge  the  design’s  trustworthiness.  A 
questionable  design  is  considered  to  be  any  design  that  has  passed 
through  an  untrusted  location  within  the  supply  chain  (e.g. 
untrusted  foundry.)  This  paper  will  review  the  parsing  of  the 
Design  Integrity  (DI)  into  five  analyzable  domains  and  discuss  the 
measuring  techniques  for  each.  A  novel  method  for  quantifying 
the  quality  of  the  reference  will  be  proposed  and  reviewed.  The 
DI  Analysis  will  then  be  applied  to  five  test  cases  in  order  to 
evaluate  and  rank  their  respective  integrities. 

II.  Quantifying  Design  Integrity 

Evaluating  Design  Integrity  can  provide  valuable  insight  into 
vetting  the  trustworthiness  of  a  design.  The  integrity  of  a  design 
can  be  defined  as  the  amount  of  deviation  observed  in  a  one-to- 
one  mapping  of  the  questionable  design  to  a  reference  profile. 
These  deviations  can  be  caused  by  carelessly  inserted  faults, 
manufacturing  flaws,  or  embedded  Trojan  circuitry.  In  essence, 
we  are  seeking  an  answer  to  the  question,  “Does  the  design 
reliably  operate  the  way  that  it  was  intended  to  without  any 
anomalous  behavior ?”  Highest  design  integrity  therefore  consists 
of  minimal  deviation  from  the  original  specification.  Lowest 
integrity  is  indicative  of  high  deviation. 
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A.  Parsing  Design  Integrity  into  Sub-Domains 

In  order  to  increase  the  insight  into  the  design,  we  parse  the 
design  into  five  character  sub-domains:  Logical  Equivalence, 
Signal  Activity  Rate,  Structural  Architecture,  Functional 
Correctness,  and  Power  Consumption.  By  evaluating  each  sub- 
domain,  one  acquires  greater  resolution  into  the  design’s 
characteristics  from  multiple  viewpoints.  Figure  1  illustrates 
conceptually  the  parsing  of  the  design  into  the  five  sub-domain 
profiles.  From  here,  both  the  expected  and  actual  properties  in 
each  domain  can  be  compared  and  the  deviation  between  the  two 
measured  by  a  technique  pertinent  to  the  reference  quality, 
amount  of  overdesign,  and  the  domain  being  analyzed. 


Figure  1  -  Parsing  Design  into  Sub-domain  Profiles 

Once  all  of  the  sub-domains  are  analyzed,  their  normalized 
deviation  measurements  can  be  aggregated  together  to  arrive  at 
the  Design  Integrity,  D1 ,  measure  expressed  as  Equation  (1)  for 
the  questionable  design.  Since  each  measurement  is  normalized, 
the  different  weights  of  each  domain  is  accounted  for  by  the 
domain  weight  factor,  yft,  which  takes  the  non-uniform  nature  of 
the  aggregated  domains  into  account.  In  this  work,  yft  was 
evaluated  as  uniform  across  all  components. 

n 

DI  =  Design  Integrity  =  z  Pi^i  (1) 

i-l 

where  Tt  =  normalized  domain  measure  technique 
and  pj  =  weighting  for  JJ.  (  (J£  =  1) 
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Figure  2  -  Domain  Specific  Integrity  Scale 
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Figure  3  -  Aggregated  Design  Integrity  Scale 

Figure  2  displays  the  normalized  DI  scale  used  for  a  single 
sub-domain.  Figure  3  represents  the  scale  for  the  aggregation 
of  all  five  sub-domains  as  determined  by  Equation  (1). 

B.  Logical  Equivalence  Integrity  Domain 

The  Logical  Equivalence  integrity,  LE integrity ,  assesses  the 
degree  to  which  the  logic  state  points  of  the  design  in  question 
map  to  the  reference  design.  The  process  verifies  the  Boolean 
logic  equivalence  of  a  given  design  at  the  same  or  different  levels 
of  abstraction  (e.g.  RTL-to-Gate)  by  injecting  test  vectors  to 
stimulate  every  logic  state  in  the  design.  Comparison  key  points 
are  defined  to  map  a  connection  between  the  reference  and 
questionable  design.  Each  key  point  state  is  evaluated  as  logically 


equivalent  or  non-equivalent.  The  equivalence  check  identifies 
the  points  between  the  two  that  are  not  equivalent,  raising  the 
concern  for  and  identifying  the  location  of  the  inserted  error. 
Equation  (2)  and  Equation  (3)  are  expressions  for  the  Equivalent 
Points  and  Total  Comparison  Points  respectively. 
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There  are  m  Total  Comparison  Points,  bi  is  a  binary  value  that 
evaluates  if  the  comparison  key  point,  Pi,  was  found  to  be 
equivalent.  For  cases  where  it  was  not  equivalent,  bt  =  0.  For  the 
case  of  an  equivalent  point,  bi=  1 .  The  utilization  factor  of  each 
Pi  is  represented  as  ay  and  gives  more  weight  to  higher  utilized 
points  and  less  weight  to  less  utilized  points.  Finally,  the  metric 
LE Megnty  is  expressed  as  Equation  (4). 

C.  Signal  Activity  Rate  Domain 

The  Signal  Rate,  SR,  defines  the  number  of  times  the 
evaluated  element  changes  state  over  the  duration  of  a  given  test 
scheme  and  is  expressed  in  units  of  millions  of  transitions  per 
second  (Mtr/s)  [4].  This  effectively  quantifies  the  activity  rate  or 
utilization  of  the  analyzed  signal  and  can  be  used  to  measure  the 
deviation  away  from  the  expected  activity  of  the  design. 
Equation  (5)  determines  the  signal  rate  for  element  i  where  fcLK  is 
the  clock  frequency  and  ay  is  the  utilization  or  toggle  rate 
percentage  of  the  element. 

^  =  fdK  Gi  (5) 


The  average  signal  rate  for  both  the  expected  and  actual  designs 
is  expressed  in  Equation  (6).  SR  is  divided  into  data,  input/output 
(IO),  and  logic  components  with  a  total  signal  quantity  of  a,  b,  c 
respectively  for  each.  This  can  be  determined  for  both  the  actual 
and  expected  and  applied  to  Equation  (7)  for  the  deviation 
distance.  Equation  (8)  is  an  expression  for  the  normalized 
SRintegrity • 
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D.  Power  Consumption  Domain 

The  Power  Consumption  domain,  P integrity,  measures  how 
closely  the  questionable  design  aligns  to  the  original  reference 
from  a  power  perspective.  Namely,  for  a  given  test  scheme,  how 
far  does  the  power  consumed  by  the  actual  design  instantiation 
deviate  away  from  the  expected  design?  Equation  (9)  expresses 
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the  expected  and  actual  power  consumptions,  P expected  and  P actual, 
at  a  power  source  point,  /,  in  the  design.  Each  power  source  point 
can  be  added  together  to  arrive  at  a  total  power  consumption  for 
both  the  expected  and  actual  power  consumptions.  The  difference 
between  the  two  can  be  represented  as  APdtst  and  expressed  as 
Equation  (10).  The  final  P integrity  is  determined  by  Equation  (11). 
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where  n,  m  =  total  source  points  for  the  questionable  and  reference 
designs  respectively 
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E. 


r expected 

Functional  Correctness  Domain 
The  Functional  Integrity,  Fintegrity ,  is  evaluated  by  observing 
the  number  of  errors  that  occur,  Observed,  for  a  given  verification 
test  scheme  and  can  be  expressed  as  Equation  (12).  TP  total  is  the 
total  verification  test  points  used  for  verifying  the  design 
functionality.  £{ observed  is  the  number  of  error  cases  accumulated 
from  the  test  scheme. 
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The  verification  test  scheme  is  designed  to  stimulate  both  the 
original  reference  and  actual  design  so  the  functionality  of  both 
designs  can  be  compared.  Test  schemes  range  from  exhaustive 
testing  to  ones  that  only  provide  comer  and  basic  functional 
coverage.  For  every  test  where  the  actual  design  does  not 
match  the  expected  result,  an  error  is  observed.  Fintegrity  can 
then  be  expressed  as  the  ratio  of  successful  tests  (i.e.  Expected 
Result  equals  Actual  Result)  to  the  total  tests  made. 

F.  Structural  Analysis  Domain 

The  Structural  Analysis  looks  at  the  architecture 
components  of  the  design  that  are  generated  once  the  design 
has  been  synthesized  into  a  gate  level  netlist.  For  an  FPGA, 
when  the  synthesis  process  is  executed,  the  design  Nets  and 
Leaf  Cells  are  represented  hierarchically  as  subcomponent 
architectures.  As  such,  these  become  the  points  of  comparison 
for  identifying  any  deviation  from  the  expected  structure. 
Equation  (13)  and  (14)  determine  the  number  of  extra  or 
removed  Nets  and  Leaf  Cells  respectively  for  an  evaluated 
architecture  component  i. 

xt  ~  \Eetsexpected  —  Netsactuai  |  (13) 

Yi  =  |  Cells  expected  ~  ^e^sactual\  (14) 

The  modified  Nets  and  Cells  can  then  be  represented  as  a  ratio 
against  the  total  Nets  and  Cells  to  arrive  at  the  Structural 
Integrity,  Sintegrity ,  expressed  in  Equation  (16).  In  order  to 
maintain  the  resolution  of  the  modified  circuits  from  getting 
washed  out  in  a  large  design,  only  the  architectures  that  show 
a  modification  to  the  Nets  or  Leaf  Cells  are  considered; 
therefore  SAX.  ^  0  and  SAYi  ^  0. 


Av[  5ax, 

m  r 

[^i  expected. 

\Xi_expected_ 

(15) 


c.  —  i  . 

° integrity  ~  ± 


■  AS 


(16) 


where  n,  m  =  number  of  modified  architectures  evaluated 
for  Nets,  Leaf  Cells  respectively  (SAX.  =/=  0  and  Sa Yi  ^0) 


III.  Test  Case  to  Evaluate  Design  Integrity 


A.  Floating  Point  Adder  System 

Several  test  cases  were  setup  in  order  to  demonstrate  the 
usefulness  of  the  DI  metric.  Figure  4  shows  a  block  diagram  of 
the  test  system,  comprised  of  two  Fixed  Point  Converters,  a 
Floating  Point  Adder,  and  an  Output  Buffer.  The  system  allows 
two  12-bit  fixed  point  inputs  to  be  converted  into  single  precision 
IEEE  754  Standard  Floating  Point  Format.  The  two  values  are 
then  added  together  and  the  result  observable  at  the  system  output. 
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Figure  4  -  Test  Case  Block  Diagram 


The  system  was  corrupted  with  the  addition  of  several  errors 
ranging  from  Stuck- At  faults  to  well-hidden  malicious  Trojans. 
This  created  a  spectrum  of  errors  with  varying  payloads  (i.e. 
damage  capability)  to  mimic  adversarial  tampering.  Table  1 
presents  details  of  each  error  as  well  as  the  insertion  location 
and  activation  mechanism  (i.e.  trigger.) 


Test  Article 

Error  Location 

Error  Trigger 

Description 

No  Error  TA 

None 

None 

No  malicious  circuitry  added  to 
design 

Error  1  TA 

Output  Buffer 

Time  Bomb  with 

Counter 

Denial  of  Service  attack  launched 
once  pre-set  time  count  is  met 

Error  2  TA 

Output  Buffer 

Counter  Trigger 

Slows  down  performance  through 
counter  delays 

Error  3  TA 

Top  Module 

Siphon  Enable 

Data  is  siphoned  to  unmonitored 
port  when  requested 

Error  4  TA 

Fixed  to  IEEE754 

Conversion 

No  Trigger 

30th  bit  of  converter  output  stuck 
at  logic  HIGH 

Table  1  -  Description  of  Errors  Inserted  into  Test  System 

Table  2  presents  the  results  of  the  analysis  applied  to  each  of  the 
test  article  designs.  Each  of  the  domains  are  evaluated  on  a  [0, 1] 
scaling  and  are  marked  with  a  color  indicative  of  the  DI  scale 
shown  in  Figure  2.  The  Design  Integrity  is  consistent  with  the 
scale  of  Figure  3.  One  can  see  that  the  integrity  for  the  No  Error 
TA  was  highest  followed  by  the  Error  4  TA.  Errors  1  and  2  TAs 
were  quantified  with  the  lowest  integrities.  Based  on  the  analysis, 
the  DI  metric  shows  measurable  differentiation  between  all  five 
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of  the  test  cases  and  lends  itself  to  the  ranking  of  each  test  article  0.20  which  is  significantly  lower  than  the  Reference  1  Trust 
in  the  order  of  highest  to  lowest  trust.  Measure  of  0.88. 


Table  2  -  Design  Integrity  Results  for  Test  System 

B.  Quantifying  the  Reference  Quality 

One  question  that  intuitively  rises  when  investigating  the 
integrity  analytics  revolves  around  the  quality  of  reference  being 
utilized  in  the  analysis.  As  such,  formulating  a  metric  for 
quantifying  the  reference  quality,  Rq,  and  correlating  it  to  the 
obtained  DI  metric  allows  one  to  place  higher  or  lower  confidence 
in  the  DI  measures.  It  also  lends  itself  to  being  used  for  comparing 
different  reference  types  and  ranking  one  against  another  in  terms 
of  usefulness.  Reference  quality  is  determined  by  Equation  (17) 
where  n  is  the  number  of  integrity  domains  the  reference  can 
evaluate  and  A  the  total  possible  domains  to  evaluate. 

Rq  —  ~  ,  where  0  <  Rq  <  1  and  n  <  N  (17) 

Table  3  shows  five  different  references  that  were  used  in  the  DI 
analysis  and  how  they  were  scored.  Rq  can  be  used  in  conjunction 
with  the  DI  metric  to  arrive  at  a  final  design  Trust  Measure 
expressed  in  Equation  (18)  that  is  indicative  of  the  confidence  one 
can  have  in  the  insights  afforded  by  the  DI  metric.  Equation  (19) 
represents  the  normalized  DI  used  to  bring  any  number  of 
domains  evaluated  into  the  [0, 1]  scale  system. 


Reference  No. 

Analyzable  Domains 

Reference 

Description  and  Format 

P 

F 

SR 

LE 

S 

Quality  (R0) 

Reference  1 

5.00 

Synthesizable  Behavioral  Design  (VHDL) 

Reference  2 

2.00 

Datasheet  Specification  (MS  Word) 

Reference  3 

1.00 

Executable  Specification  (MATLAB) 

Reference  4 

3.00 

Datasheet  with  Executable  Specification 
(MATLAB/MS  Word) 

Reference  5 

4.00 

Synthesized  Netlist  (Verilog) 

Table  3  -  Description  of  References 


Test  Article  N  n 


No  Error  TA  5  5 

Error  1 TA  5  5 

Error  2  TA  5  5 

Error  3  TA  5  5 

Error  4  TA  5  5 


Reference  1  (RQ  =  1) 


Reference  2  (RQ  =  2/5) 


Reference  3  (RQ  =  1/5) 


Rq  DI  DInorm 


5.00 

1.00 

3.27 

0.65 

2.95 

0.59 

4.38 

0.88 

4.40 

0.88 

1.00 

2 

0.4 

2.00 

1.00 

0.40 

0.65 

2 

0.4 

1.71 

0.85 

0.34 

0.59 

2 

0.4 

1.32 

0.66 

0.26 

0.88 

2 

0.4 

1.59 

0.79 

0.32 

0.88 

2 

0.4 

1.91 

0.96 

0.38 

0.2  1.00  L00020 
0.2  0.00  ■ 

0.2  0.13  | 

0.2  1.00  1.00  (UOJ 

0.2  0.50  0.50  j 


Test  Article 

N 

n 

Reference  4  (RQ  =  3/5) 

Rq  DI  DInorm  TM 

n 

Reference  5  (RQ  = 
Rq  DI  DInorir 

4/5) 

,  TM 

No  Error  TA 

5 

3 

0.6 

3.00 

1.00 

0.60 

4 

0.8 

4.00 

1.00 

0.80 

Error  1  TA 

5 

3 

0.6 

1.71 

0.57 

0.34 

4 

0.8 

3.27 

0.82 

0.65 

Error  2  TA 

5 

3 

0.6 

1.45 

0.48 

0.29 

4 

0.8 

2.82 

0.70 

0.56 

Error  3  TA 

5 

3 

0.6 

2.59 

0.86 

0.52 

4 

0.8 

3.38 

0.85 

0.68 

Error  4  TA 

5 

3 

0.6 

2.41 

0.80 

0.48 

4 

0.8 

3.90 

0.98 

0.78 

Trust  Scaling 


Highest  Trust 


Lowest  Trust 


Table  4  -  Comparison  of  Different  References  Types 

IV.  Conclusion  and  Future  Work 

This  paper  proposed  several  techniques  for  evaluating  a 
design’s  integrity  by  looking  at  five  different  characteristic 
domains  of  the  design  and  then  aggregating  their  measured 
deviations  from  expected  characteristics  together  to  arrive  at  a 
single  value  DI  metric.  A  novel  method  for  quantifying 
references  was  also  presented  that  considers  the  quality  of  the 
reference  being  used  in  the  DI  Analysis.  Quality  of  the 
reference  is  a  way  to  capture  amount  of  overdesign  into  the  DI 
metric.  A  final  Trust  Measure  indicative  of  the  design’s 
integrity  and  the  confidence  provided  by  the  reference  quality 
was  achieved.  The  major  takeaway  from  this  work  is  that  one 
can  now  quantifiably  indicate  that  a  design  has  poorer  or  higher 
integrity  and  measurably  present  how  far  it  has  deviated  away 
from  the  expected  characteristics.  By  establishing  reference 
quality  metrics,  the  quality  of  the  DI  value  obtained  from 
different  references  can  be  compared  and  different  references 
ranked  according  to  their  utility. 

The  development  of  trust  metrics  is  an  iterative  process  with 
future  iterations  improving  on  and  adding  to  the  previous  ones. 
New  domains  for  characterizing  the  design  need  to  be  explored 
to  increase  the  observability  into  hardware.  The  weighting 
factor,  /?*,  of  each  domain  also  remains  to  be  determined  to 
address  the  normalization  approach  taken  in  each  sub-domain. 
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Table  4  revisits  the  DI  metrics  presented  in  Table  2  and  shows 
how  the  TA  cases  would  be  evaluated  with  different  references. 
Reference  1  was  the  highest  quality  because  it  applied  to  all  five 
domains  of  the  DI  analysis.  Reference  3  was  the  lowest  quality 
lending  itself  to  be  utilized  in  only  one  domain.  The  impact  of 
reference  quality  on  quantifying  design  integrity  is  highlighted 
with  the  Error  3  TA  example.  DInorm  was  measured  as  0.88  for 
Reference  1,  but  measured  as  1.00  by  Reference  3.  This  is 
because  Reference  3  does  not  afford  the  level  of  observability  that 
Reference  1  does  into  the  design  to  track  the  deviations  caused  by 
the  error.  Based  on  this  information,  one  could  be  led  to  believe 
that  Error  3  TA  was  of  highest  trust.  The  Trust  Measure  however 
accounts  for  the  poor  reference  quality  and  adjusts  the  scoring  to 
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